Predicting the efficacy of issues detected with machine executed digitized intellectual capital

ABSTRACT

A digitized Intellectual Capital (IC) system obtains code modules configured to detect one or more issues in a computing system. The IC system selects from the code modules to generate a first set of code modules based on a corresponding value metric. The corresponding value metric for each code module in the first set of code modules is higher than a predetermined threshold. The IC system also samples from the remainder of the code modules unselected for the first set of code modules to generate a second set of code modules. The IC system runs the first set of code modules and the second set of code modules to detect the one or more issues and updates the corresponding value metric for at least one code module.

TECHNICAL FIELD

The present disclosure relates to management of intellectual capital,especially with code modules representing the intellectual capital.

BACKGROUND

Intellectual Capital (IC) can be captured and digitized in the form ofcode modules written to detect issues in a computing system. The IC codemodules are typically written to detect a particular issue in aparticular environment. The issue in the computing device may be relatedto error conditions, potential error conditions, and/or best practicesin configuration. Typically, an IC engine runs multiple IC code modulesagainst a computing system to search for multiple issues at a time. NewIC code modules are typically added to detect new types of issues, or todetect issues in new types of computing systems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram of an IC system for detectingissues in a computing system, according to an example embodiment.

FIG. 2 illustrates the life cycle of an IC code module, according to anexample embodiment.

FIG. 3 illustrates the application of IC code modules to a computingsystem, according to an example embodiment.

FIG. 4 is a flowchart illustrating operations performed at an IC systemprocessor to detect issues in a computing system and update the value ofIC code modules, according to an example embodiment.

FIG. 5 illustrates a simplified block diagram of a device that may beconfigured to perform the methods presented herein, according to anexample embodiment.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

A computer implemented method is provided for managing digitized IC. Themethod includes obtaining a plurality of code modules configured todetect one or more issues in a computing system. The method alsoincludes selecting from the plurality of code modules to generate afirst set of code modules based on a corresponding value metric. Thecorresponding value metric for each code module in the first set of codemodules is higher than a predetermined threshold. The method furtherincludes sampling from a remainder of the plurality of code modulesunselected for the first set of code modules to generate a second set ofcode modules. The method includes running the first set of code modulesand the second set of code modules to detect the one or more issues andupdating the corresponding value metric for at least one code module inthe plurality of code modules.

Example Embodiments

Digitized Intellectual Capital (IC), or machine executable issuedetection, extends the reach of human knowledge more broadly in anetwork environment by enabling the automated identification of presentand potential problems that could impact the availability of networkservices. Converting human knowledge or even machine generatedknowledge, such as anomaly detection, is context-specific and contextchanges as a function of product life cycle and changes in the network.Left unchecked the amount of available IC will continue to grow,requiring more time and greater resources to execute. The techniquespresented herein provide a method and system for determining therelative value of available IC, enabling an operator to predict whetherthe return on investment justifies retaining an IC module as part of theworking set of IC. This approach combines a number of factors having todo with the IC execution as well as customer consumption of the resultsto identify IC that should be decommissioned and no longer executed.

The creation and execution of human-created, machine-consumable IC codemodules is associated with various business costs. Typical IC codemodule creation involves a user having to identify issues that may becandidates for machine detection. Once identified, the issue may betriaged by a user who determines whether the issue can be detected withthe available output, as well as determining a form (e.g., on a device,controller, or in the cloud) for capturing and executing a signature ofthe issue. The code module for the detection and remediation logic maythen be written, classified, tested, published and deployed. Once,deployed, the code module may be executed periodically (e.g.,proactively) or on demand (e.g., in support of specific tasks).Typically, a code module is executed to determine whether it isapplicable for a particular context (e.g., device, operating systemversion, role in the network, etc.) for at least some part of thesignature detection (e.g., accessing the signature or executingprecondition logic). If the code module passes the preconditions to beapplicable to the particular context, the detection logic in the codemodule may be executed, triggering additional data collection from thecomputing system.

As new devices and elements are added to a system, and the configurationof existing devices and elements are updated (e.g., with new operatingsystems) or phased out, new IC code modules may be created to detectissues. Additionally, existing IC code modules may be updated to adaptto changes in the computing system, adding additional business costs tothe overall investment in the digitization of the IC system.

Creating and maintaining tens of thousands of IC code modules, as wellas executing those code modules on potentially millions of devicesrequires significant human investment, tooling investment, and computingresources. If the IC repositories continue to grow as the development ofcomputing systems continues, resources may be spent oncreating/maintaining code modules that no longer provide any value tothe customer. The techniques presented herein provide a system forautomatically identifying code modules that should no longer be includedin the deployed set of code modules, eliminating additional businesscosts of making the decision to retire or further maintain specific codemodules. Additionally, the techniques presented herein may provide aprediction of when a particular IC code module may lose business value,enabling the scheduled removal of IC code modules.

Referring now to FIG. 1, a system 100 is shown for detecting issues in acomputing system 110. The computing system 110 may include one or morenetwork devices 112, one or more computing devices 114, and/or one ormore Internet of Things (IoT) devices 116. In one example, the networkdevices 112 may include switches, routers, gateways, or other networkelements configured to transfer data. In another example, the computingdevices 114 may include servers, personal computers, virtual machines,or other computing devices configured to process data. In a furtherexample, the IoT devices 116 may include appliances, sensors, or otherembedded devices.

The computing system 110 is connected to an IC system 120 that isconfigured to detect issues in the computing system 110 using IC codemodules. The individual code modules are stored in an IC repository 130.In one example, the IC repository 130 may be one or more memory devicesconfigured to store electronic data, such as the code modules, and mayinclude multiple devices for redundancy, reliability, speed, and/orbandwidth. The code modules in the IC system 120 are run on an IC engine140. In one example, the IC engine may be one or more processorsconfigured to obtain data from the computing system 110 and run one ormore code modules from the IC repository 130.

The IC system 120 also includes an IC selection logic 150 that enablesthe IC system 120 to select code modules from the IC repository 130 torun on the IC engine 140. In one example, the IC selection logic 150selects code modules that are applicable and most likely to providevalue in detecting issues in the computing system 110. The IC system 120includes an issue resolution logic 160 that enables the IC system 120 todetect whether the issues detected by the code modules are resolved inthe computing system 110. The IC system 120 further includes ICdecommissioning logic 170 that enables the IC system 120 to remove ICcode modules that no longer provide value in finding issues in thecomputing system 110.

As the number of code modules in the IC repository 130 grows, and thenumber and configuration of the devices in the computing system 110changes, the data and resources needed to execute all of the codemodules grows. The IC system 120 provides a means for properly managingand culling the code modules based on whether the code modules continueto provide value to the customer (e.g., the operator of the computingsystem 110). The IC system 120 identifies metrics and metadata todetermine present customer value, and predicts when specific codemodules may no longer be valuable. In one example, the relevant metricsfor value may include the number of executions of a code module, thenumber of detections by a code module, the number of times the issuedetected by a code module was ignored by the customer, and/or the numberof times the issue was resolved in the computing system 110. In anotherexample, the metadata for customer value may include IC classification,product specificity and life cycle, customer preferences, and/or therole of the devices.

Referring now to FIG. 2, an example of the life cycle of IC codemodule(s) 210 is shown to highlight the value to the customer. The ICcode modules 210 are developed and stored in the IC repository 130. Thecreation and maintenance of the IC code modules 210 constitute resourceexpenditure to increase the coverage of the code modules or to preventthe decrease of coverage of the code modules. The IC system 120 collectsdata from the computing system 110 and uses the IC engine 140 to applyselected code modules from the IC repository 130 to the data from thecomputing system 110 to detect issues in the computing system 110.Running the IC engine 140 (e.g., collecting data, selecting codemodules, running code modules) constitutes additional resourceexpenditure (e.g., to obtain cloud-provided storage and processingpower). Running the IC engine 140 also produces IC results, such asissue detection, alerts, and recommendations for remediation.

The IC system 120 may choose some or all of the IC results to send tothe operator 220 of the computing system 110 or an automated system runby the operator 220 of the computing system 110. In one example, theautomated system run by the operator 220 may include a networkmanagement system with logic to react and respond to issues detected ina computer network (e.g., computing system 110) by the IC system 120.Receiving and parsing the IC results may impose a cognitive load onhuman operators or resource expenditure on of automated systems. Theoperator 220 may apply changes to the computing system 110 based on therecommendations in the IC results, which results in future changes inthe data provided to the IC engine 140 in future runs of selected codemodules from the IC repository 130. Additionally, the IC system 120 mayuse the data obtained from the computing system, or obtained directlyfrom the operator 220, to select IC code modules 230 to decommission andremove from the IC repository 130.

The value of the IC system 120 may be viewed as a difference between theexpenditure in running the IC system 120 (e.g., maintenance of codemodules, running code modules, cognitive load in parsing results, etc.)and the improvements to the computing system (e.g., in availability,time-to-resolution, etc.). This value may be quantified by comparing KeyPerformance Indicators (KPIs) before and after the IC system 120 isdeployed on the computing system 110 or comparable systems.

In one example, a number of factors may be applied to determine a valuemetric to balance with the resource expenditure of running a code moduleon the IC system 120. The value metric of a code module may be based onthe applicability of the code module (e.g., whether thesoftware/hardware configurations required by the code module are presentin the computing system 110), running the code module (e.g., the numberof times a code module is run by the IC engine 140), detecting issues(e.g., the number of times the code module detects an issue),impressions to the operator (e.g., the number of times the code modulegenerates a result that is presented to the operator 220), andremediating issues (e.g., the number of times a subsequent run of thecode module detects the issue is no longer present in the computingsystem 110).

Detections that correlate strongly with rapid changes and clearing of anunderlying issue may be a good indicator of utility and value. Codemodules with high detection KPIs are typically prioritized over codemodules that are not applicable to the computing system 110, or thatfind few issues in the computing system 110, or that find issues thatare rarely/never fixed by the operator 220. However, some issues withthe computing system 110 may not be adequately captured by code modulesthat are prioritized by repeatedly detecting and clearing issues.

In one example, the computing system 110 may have an intermittent issuethat appears and disappears without operator involvement. Code modulesthat detect this type of issue may be prioritized by metrics such asprediction error of the time to fix, recurrence, age/detection rate,and/or internetwork entropy. Other code modules may detect extremelyrare issues, but the impact of the issue justifies the continuedpresence of these code modules in the IC repository. The IC system 120may use specific metadata to prioritize these high-impact code modulesabove the value metric that would be assigned solely based on thestatistical value estimation.

Referring now to FIG. 3, a flowchart illustrates an example of the ICsystem selecting IC code modules and providing feedback on the value ofindividual code modules. The IC system compares the code modules in theIC repository 130 with the configuration of the computing system 110 toobtain a set of IC code modules 310 that are applicable to the computingsystem. For instance, the IC system may inspect the current state of thecomputing system 110 (e.g., device hardware and software) and providecode modules that are designed to detect issues in particular devicesand/or software versions that are present in the computing system 110 togenerate the set of applicable code modules 310.

The set of applicable code modules 310 is provided to an exploit branch320 and an explore branch 325 to select a set of code modules thatprovide the most value to the operator of the computing system 110. Inthe exploit branch 320, a regressor 330 collects IC KPIs 332 (e.g.,applicability scope, a history of runs, a history of detections, ahistory of impressions, a history of remediations, etc.), as well as ICmetadata 334 (e.g., the age or specific priority category of the codemodule) to generate a value metric 336 for each code module. In oneexample, the value metric may be a quantified as a probability that anissue will be found by the associated code module and remediated by theoperator of the computing system 110. In another example, the valuemetric may be represented by a vector presenting each uncertaintyseparately as either a probability (e.g., 60% chance to find an issuethat has been fixed 75% of the time in the past) or as a count (e.g.,the code module will find N issues, and M issues will be fixed).

A selector 338 in the exploit branch 320 selects the code modules mostlikely to detect issues and provide value to the operator of thecomputing system. In one example, the selector 338 may select all of thecode modules with a value metric above a predetermined threshold.Alternatively, the selector 338 may select the code modules associatedwith the top percentage (e.g., the top 60%) of the value metrics. Theexploit branch 320 is configured to select code modules based on thelikelihood that the code module will detect an issue that will beremediated by the operator of the computing system 110 and add theselected code modules to the run set of IC code modules 340. In anotherexample, the exploit branch 320 may be configured to include a selectionof code modules that detect issues that provide a significant impact ona particular computing system of a particular operator, even when themeasured IC KPIs 332 are relatively low.

The explore branch 325 adds to the run set of IC code modules 340 byusing a sampler 350 to sample applicable code modules that have not beenselected by the selector 338 based on an accumulated value metric 352and KPI entropy 354. The sampler 350 is partly stochastic in samplingfrom the remaining code modules, but the odds of selection may not beequal for each code module. The explore branch maintains the accumulatedvalue metric 352 for any code module that was not selected by theselector 338 or the sampler 350. The accumulated value metric 352 booststhe odds of being sampled by the sampler 350 in future run sets.

The IC engine 140 runs the run set of IC code modules 340 to generate ICresults 360. Based on the IC results 360, the IC system may provide theexploit branch 320 with a KPI update 370 for the code modules that wereincluded in the run set of IC code modules 340, including the codemodules added by the explore branch 325.

Depending on the IC code modules and the computing system 110, the valuemetric 336 calculated by the regressor 330 may include a variable amountof uncertainty. The explore branch 325 is configured to sample codemodules that have not been selected by the exploit branch 320, but whichmay still provide value to the operator of the computing system 110.Additionally, the explore branch 325 may provide training data for theregressor 330 to better evaluate code modules to determine an accuratevalue metric 336.

In one example, the regressor 330 may be based on a machine learningmodel that is iteratively trained by the IC results 360 and the KPIupdate 370. Including training data from the explore branch 325 enablesthe IC system to train the regressor 330 in a self-supervised manner,and makes the process economical without expert data curation. However,expert data curation may provide faster convergence for a machinelearning regressor 330. A machine learning regressor 330 may be based ona Generalized Additive Model (GAM) or a Generalized Linear Model (GLM)for situations in which transparency is important. Alternatively, neuralnetwork models (e.g., Deep Neural Network, Recursive Neural Network) maybe used for the regressor 330 for situations in which additionalpredictive power and model capacity provide a benefit.

The goal of the explore branch 325 may be two-fold. Initially, theregressor 330 may not be adequately trained to accurately predict whichcode modules will have a high value metric 336, and the sampler 350 mayprovide the majority of the run set of IC code modules 340. As the codemodules are run multiple times, the KPI updates 370 may providesufficient training for the regressor 330 to enable the selector 338 topopulate a more significant portion of the run set of IC code modules340.

Additionally, the explore branch 325 may compensate for predictionerrors in the exploit branch 320. In other words, the selector 338 maynot select a particular code module that would resolve an uncommon issuein the computing system 110. The explore branch 325 provides achance/possibility that the particular code module will be pickedstochastically, which would allow the KPI update 370 to retrain theregressor 330. Retraining of the regressor 330 provides for a moreaccurate value metric 336, which enables the selector 338 to provide amore accurate prediction of code modules that are likely to providevalue in finding issues in the computing system 110. Code modulesconfigured to detect issues that are truly random will almost certainlybe added to the run set of IC code modules 340 through the explorebranch 325, since the value metric of a code module for a random eventis likely to remain relatively low.

Operating the IC system described herein over time on a number ofnetworks may produce data that is useful to automatically decommissionlegacy code modules that no longer provide value to the operator of thecomputing system 110. In one example, the IC system may automaticallydecommission legacy code modules with a value metric that is uniformlydeclining over time. The threshold for decommissioning a code module maybe driven by the bandwidth provided to the explore branch 325. If the ICsystem does not provide computing power to allow the explore branch toinject enough code modules to provide training data for the exploitbranch 320, then the IC system either needs to allocate additionalcomputing power to the IC engine 140 (i.e., enlarge the run set of ICcode modules 340 and run more code modules from the explore branch) orthe corpus of IC code modules needs to be reduced.

In another example, multiple IC code modules may be treated as relatedto conserve processing resources. Given the finite amount of resourcesusable in IC code module execution, each IC code module may be graded(e.g., in the exploit branch 320) with a binary classifier (i.e., aboveor below a predetermined threshold) based on the metadata and KPIstatistics for the IC code module. Grouping similar IC code modulesenables the IC system 120 to apply the binary classification of a singleIC code module to a suite of related IC code modules, rather thanindividually classify each IC code module in the suite. For instance,one IC code module may be configured to detect a particular networkissue for one version of an operating system on a router. Related ICcode modules may include IC code modules configured to detect the samenetwork issue on the same router that is running a newer version of theoperating system. Alternatively, related IC code modules may be groupedbased on the similarity of the issues that each IC code module isconfigured to detect.

Applying the same classification to multiple IC code modules enables theIC system 120 to operate without constant data curation and tagging fornew IC code module, because each new IC code module may be related to anexisting IC code module that has already been classified. In oneexample, to address the cold start problem, the regressor 330 may betrained on a small number of expert-curated examples in which the entireIC corpus stored in the IC repository 130 is executed, and split intorelated groups to be annotated by a subject matter expert.

In a further example, with an IC engine 140 that is instrumented toprovide near-real-time data, another machine learning approach may be touse a reinforcement learning model to directly maximize the valuemetric. The reinforcement learning model may use resource expenditure asa negative outcome and KPI statistics as a positive outcome. Thereinforcement learning action space may be directed to making decisionsto maintain and run each IC code module. The reinforcement learningstate space may include the statistics of code modules, metadata of thecode modules, as well as a window of detections by IC code modules overtime.

Referring now to FIG. 4, a flowchart illustrates operations performed byan IC system (e.g., IC system 120) in a process 400 for managing codemodules to detect issues in a computing system. At 410, the IC systemobtains a plurality of code modules that are configured to detect one ormore issues in a computing system. In one example, the code modules maybe machine consumable code modules that detect issues in a computingsystem and provide remediation recommendations to the operator of thecomputing system.

At 420, the IC system selects code modules from the plurality of codemodules based on a corresponding value metric for each code module. Theselected code modules generate a first set of code modules that will berun by the IC system. In one example, the value metric may be based onstatistical KPI data and/or metadata associated with each code module.The IC system may select code modules for the first set of code modulesbased on the corresponding value metric exceeding a predeterminedthreshold.

At 430, the IC system samples from the remaining code modules that havenot been selected for the first set of code modules and generates asecond set of code modules. In one example, the second set of codemodules may be sampled by way of a stochastic process based on anaccumulation metric that boosts the probability of a code module beingincluded in the second set of code modules. The accumulation metric maybe based on the value metric associated with the code module and thenumber of times the code module has not been selected to be run by theIC system.

At 440, the IC system runs the first set of code modules and the secondset of code modules to detect one or more issues in the computingsystem. In one example, running the first set of code modules and thesecond set of code modules provides feedback that one or more issues arepresent in the computing system, as well as potential remediation stepsfor resolving the issues. Based on the results of running the first setof code modules and the second set of code modules, the IC systemupdates the value metric for at least one code module at 450. In oneexample, the IC system may increase the value metric for a code modulethat detected an issue and presented the issue to an operator of thecomputing system. The IC system may further increase the value metric ofa particular code module based on a detection that the computing systemhas resolved an issue previously detected by that particular codemodule.

Referring to FIG. 5, FIG. 5 illustrates a hardware block diagram of acomputing device 500 that may perform functions associated withoperations discussed herein in connection with the techniques depictedin FIGS. 1-4. In various embodiments, a computing device, such ascomputing device 500 or any combination of computing devices 500, may beconfigured as any entity/entities as discussed for the techniquesdepicted in connection with FIGS. 1-4 in order to perform operations ofthe various techniques discussed herein.

In at least one embodiment, the computing device 500 may include one ormore processor(s) 502, one or more memory element(s) 504, storage 506, abus 508, one or more network processor unit(s) 510 interconnected withone or more network input/output (I/O) interface(s) 512, one or more I/Ointerface(s) 514, and control logic 520. In various embodiments,instructions associated with logic for computing device 500 can overlapin any manner and are not limited to the specific allocation ofinstructions and/or operations described herein.

In at least one embodiment, processor(s) 502 is/are at least onehardware processor configured to execute various tasks, operationsand/or functions for computing device 500 as described herein accordingto software and/or instructions configured for computing device 500.Processor(s) 502 (e.g., a hardware processor) can execute any type ofinstructions associated with data to achieve the operations detailedherein. In one example, processor(s) 502 can transform an element or anarticle (e.g., data, information) from one state or thing to anotherstate or thing. Any of potential processing elements, microprocessors,digital signal processor, baseband signal processor, modem, PHY,controllers, systems, managers, logic, and/or machines described hereincan be construed as being encompassed within the broad term ‘processor’.

In at least one embodiment, memory element(s) 504 and/or storage 506is/are configured to store data, information, software, and/orinstructions associated with computing device 500, and/or logicconfigured for memory element(s) 504 and/or storage 506. For example,any logic described herein (e.g., control logic 520) can, in variousembodiments, be stored for computing device 500 using any combination ofmemory element(s) 504 and/or storage 506. Note that in some embodiments,storage 506 can be consolidated with memory element(s) 504 (or viceversa), or can overlap/exist in any other suitable manner.

In at least one embodiment, bus 508 can be configured as an interfacethat enables one or more elements of computing device 500 to communicatein order to exchange information and/or data. Bus 508 can be implementedwith any architecture designed for passing control, data and/orinformation between processors, memory elements/storage, peripheraldevices, and/or any other hardware and/or software components that maybe configured for computing device 500. In at least one embodiment, bus508 may be implemented as a fast kernel-hosted interconnect, potentiallyusing shared memory between processes (e.g., logic), which can enableefficient communication paths between the processes.

In various embodiments, network processor unit(s) 510 may enablecommunication between computing device 500 and other systems, entities,etc., via network I/O interface(s) 512 to facilitate operationsdiscussed for various embodiments described herein. In variousembodiments, network processor unit(s) 510 can be configured as acombination of hardware and/or software, such as one or more Ethernetdriver(s) and/or controller(s) or interface cards, Fibre Channel (e.g.,optical) driver(s) and/or controller(s), and/or other similar networkinterface driver(s) and/or controller(s) now known or hereafterdeveloped to enable communications between computing device 500 andother systems, entities, etc. to facilitate operations for variousembodiments described herein. In various embodiments, network I/Ointerface(s) 512 can be configured as one or more Ethernet port(s),Fibre Channel ports, and/or any other I/O port(s) now known or hereafterdeveloped. Thus, the network processor unit(s) 510 and/or network I/Ointerface(s) 512 may include suitable interfaces for receiving,transmitting, and/or otherwise communicating data and/or information ina network environment.

I/O interface(s) 514 allow for input and output of data and/orinformation with other entities that may be connected to computer device500. For example, I/O interface(s) 514 may provide a connection toexternal devices such as a keyboard, keypad, a touch screen, and/or anyother suitable input and/or output device now known or hereafterdeveloped. In some instances, external devices can also include portablecomputer readable (non-transitory) storage media such as databasesystems, thumb drives, portable optical or magnetic disks, and memorycards. In still some instances, external devices can be a mechanism todisplay data to a user, such as, for example, a computer monitor, adisplay screen, or the like.

In various embodiments, control logic 520 can include instructions that,when executed, cause processor(s) 502 to perform operations, which caninclude, but not be limited to, providing overall control operations ofcomputing device; interacting with other entities, systems, etc.described herein; maintaining and/or interacting with stored data,information, parameters, etc. (e.g., memory element(s), storage, datastructures, databases, tables, etc.); combinations thereof; and/or thelike to facilitate various operations for embodiments described herein.

The programs described herein (e.g., control logic 520) may beidentified based upon application(s) for which they are implemented in aspecific embodiment. However, it should be appreciated that anyparticular program nomenclature herein is used merely for convenience;thus, embodiments herein should not be limited to use(s) solelydescribed in any specific application(s) identified and/or implied bysuch nomenclature.

In various embodiments, entities as described herein may storedata/information in any suitable volatile and/or non-volatile memoryitem (e.g., magnetic hard disk drive, solid state hard drive,semiconductor storage device, random access memory (RAM), read onlymemory (ROM), erasable programmable read only memory (EPROM),application specific integrated circuit (ASIC), etc.), software, logic(fixed logic, hardware logic, programmable logic, analog logic, digitallogic), hardware, and/or in any other suitable component, device,element, and/or object as may be appropriate. Any of the memory itemsdiscussed herein should be construed as being encompassed within thebroad term ‘memory element’. Data/information being tracked and/or sentto one or more entities as discussed herein could be provided in anydatabase, table, register, list, cache, storage, and/or storagestructure: all of which can be referenced at any suitable timeframe. Anysuch storage options may also be included within the broad term ‘memoryelement’ as used herein.

Note that in certain example implementations, operations as set forthherein may be implemented by logic encoded in one or more tangible mediathat is capable of storing instructions and/or digital information andmay be inclusive of non-transitory tangible media and/or non-transitorycomputer readable storage media (e.g., embedded logic provided in: anASIC, digital signal processing (DSP) instructions, software[potentially inclusive of object code and source code], etc.) forexecution by one or more processor(s), and/or other similar machine,etc. Generally, memory element(s) 504 and/or storage 506 can store data,software, code, instructions (e.g., processor instructions), logic,parameters, combinations thereof, and/or the like used for operationsdescribed herein. This includes memory element(s) 504 and/or storage 506being able to store data, software, code, instructions (e.g., processorinstructions), logic, parameters, combinations thereof, or the like thatare executed to carry out operations in accordance with teachings of thepresent disclosure.

In some instances, software of the present embodiments may be availablevia a non-transitory computer useable medium (e.g., magnetic or opticalmediums, magneto-optic mediums, CD-ROM, DVD, memory devices, etc.) of astationary or portable program product apparatus, downloadable file(s),file wrapper(s), object(s), package(s), container(s), and/or the like.In some instances, non-transitory computer readable storage media mayalso be removable. For example, a removable hard drive may be used formemory/storage in some implementations. Other examples may includeoptical and magnetic disks, thumb drives, and smart cards that can beinserted and/or otherwise connected to a computing device for transferonto another computer readable storage medium.

Variations and Implementations

Embodiments described herein may include one or more networks, which canrepresent a series of points and/or network elements of interconnectedcommunication paths for receiving and/or transmitting messages (e.g.,packets of information) that propagate through the one or more networks.These network elements offer communicative interfaces that facilitatecommunications between the network elements. A network can include anynumber of hardware and/or software elements coupled to (and incommunication with) each other through a communication medium. Suchnetworks can include, but are not limited to, any local area network(LAN), virtual LAN (VLAN), wide area network (WAN) (e.g., the Internet),software defined WAN (SD-WAN), wireless local area (WLA) access network,wireless wide area (WWA) access network, metropolitan area network(MAN), Intranet, Extranet, virtual private network (VPN), Low PowerNetwork (LPN), Low Power Wide Area Network (LPWAN), Machine to Machine(M2M) network, Internet of Things (IoT) network, Ethernetnetwork/switching system, any other appropriate architecture and/orsystem that facilitates communications in a network environment, and/orany suitable combination thereof.

Networks through which communications propagate can use any suitabletechnologies for communications including wireless communications (e.g.,4G/5G/nG, IEEE 802.11 (e.g., Wi-Fi®/Wi-Fi6®), IEEE 802.16 (e.g.,Worldwide Interoperability for Microwave Access (WiMAX)),Radio-Frequency Identification (RFID), Near Field Communication (NFC),Bluetooth™ mm.wave, Ultra-Wideband (UWB), etc.), and/or wiredcommunications (e.g., T1 lines, T3 lines, digital subscriber lines(DSL), Ethernet, Fibre Channel, etc.). Generally, any suitable means ofcommunications may be used such as electric, sound, light, infrared,and/or radio to facilitate communications through one or more networksin accordance with embodiments herein. Communications, interactions,operations, etc. as discussed for various embodiments described hereinmay be performed among entities that may directly or indirectlyconnected utilizing any algorithms, communication protocols, interfaces,etc. (proprietary and/or non-proprietary) that allow for the exchange ofdata and/or information.

In various example implementations, entities for various embodimentsdescribed herein can encompass network elements (which can includevirtualized network elements, functions, etc.) such as, for example,network appliances, forwarders, routers, servers, switches, gateways,bridges, loadbalancers, firewalls, processors, modules, radioreceivers/transmitters, or any other suitable device, component,element, or object operable to exchange information that facilitates orotherwise helps to facilitate various operations in a networkenvironment as described for various embodiments herein. Note that withthe examples provided herein, interaction may be described in terms ofone, two, three, or four entities. However, this has been done forpurposes of clarity, simplicity and example only. The examples providedshould not limit the scope or inhibit the broad teachings of systems,networks, etc. described herein as potentially applied to a myriad ofother architectures.

Communications in a network environment can be referred to herein as‘messages’, ‘messaging’, ‘signaling’, ‘data’, ‘content’, ‘objects’,‘requests’, ‘queries’, ‘responses’, ‘replies’, etc. which may beinclusive of packets. As referred to herein and in the claims, the term‘packet’ may be used in a generic sense to include packets, frames,segments, datagrams, and/or any other generic units that may be used totransmit communications in a network environment. Generally, a packet isa formatted unit of data that can contain control or routing information(e.g., source and destination address, source and destination port,etc.) and data, which is also sometimes referred to as a ‘payload’,‘data payload’, and variations thereof. In some embodiments, control orrouting information, management information, or the like can be includedin packet fields, such as within header(s) and/or trailer(s) of packets.Internet Protocol (IP) addresses discussed herein and in the claims caninclude any IP version 4 (IPv4) and/or IP version 6 (IPv6) addresses.

To the extent that embodiments presented herein relate to the storage ofdata, the embodiments may employ any number of any conventional or otherdatabases, data stores or storage structures (e.g., files, databases,data structures, data or other repositories, etc.) to store information.

Note that in this Specification, references to various features (e.g.,elements, structures, nodes, modules, components, engines, logic, steps,operations, functions, characteristics, etc.) included in ‘oneembodiment’, ‘example embodiment’, ‘an embodiment’, ‘anotherembodiment’, ‘certain embodiments’, ‘some embodiments’, ‘variousembodiments’, ‘other embodiments’, ‘alternative embodiment’, and thelike are intended to mean that any such features are included in one ormore embodiments of the present disclosure, but may or may notnecessarily be combined in the same embodiments. Note also that amodule, engine, client, controller, function, logic or the like as usedherein in this Specification, can be inclusive of an executable filecomprising instructions that can be understood and processed on aserver, computer, processor, machine, compute node, combinationsthereof, or the like and may further include library modules loadedduring execution, object files, system files, hardware logic, softwarelogic, or any other executable modules.

It is also noted that the operations and steps described with referenceto the preceding figures illustrate only some of the possible scenariosthat may be executed by one or more entities discussed herein. Some ofthese operations may be deleted or removed where appropriate, or thesesteps may be modified or changed considerably without departing from thescope of the presented concepts. In addition, the timing and sequence ofthese operations may be altered considerably and still achieve theresults taught in this disclosure. The preceding operational flows havebeen offered for purposes of example and discussion. Substantialflexibility is provided by the embodiments in that any suitablearrangements, chronologies, configurations, and timing mechanisms may beprovided without departing from the teachings of the discussed concepts.

As used herein, unless expressly stated to the contrary, use of thephrase ‘at least one of’, ‘one or more of’, ‘and/or’, variationsthereof, or the like are open-ended expressions that are bothconjunctive and disjunctive in operation for any and all possiblecombination of the associated listed items. For example, each of theexpressions ‘at least one of X, Y and Z’, ‘at least one of X, Y or Z’,‘one or more of X, Y and Z’, ‘one or more of X, Y or Z’ and ‘X, Y and/orZ’ can mean any of the following: 1) X, but not Y and not Z; 2) Y, butnot X and not Z; 3) Z, but not X and not Y; 4) X and Y, but not Z; 5) Xand Z, but not Y; 6) Y and Z, but not X; or 7) X, Y, and Z.

Additionally, unless expressly stated to the contrary, the terms‘first’, ‘second’, ‘third’, etc., are intended to distinguish theparticular nouns they modify (e.g., element, condition, node, module,activity, operation, etc.). Unless expressly stated to the contrary, theuse of these terms is not intended to indicate any type of order, rank,importance, temporal sequence, or hierarchy of the modified noun. Forexample, ‘first X’ and ‘second X’ are intended to designate two ‘X’elements that are not necessarily limited by any order, rank,importance, temporal sequence, or hierarchy of the two elements. Furtheras referred to herein, ‘at least one of’ and ‘one or more of’ can berepresented using the ‘(s)’ nomenclature (e.g., one or more element(s)).

In summary, the techniques presented herein manage digitized IC toproactively identify potential issues that could impact the performanceof a computing system as well as to assist introubleshooting/remediation of issues with the computing system. Thetechniques presented herein mitigate human costs and resourceexpenditures associated with updating and executing the IC code modulesby aging out IC code modules that may no longer be applicable or providebusiness value. The techniques enable the IC system to predict when ICcode modules will no longer provide value, enabling businesses toproactively plan and remove the ongoing maintenance costs of these ICcode modules.

In one form, a method is provided for managing digitized IC. The methodincludes obtaining a plurality of code modules configured to detect oneor more issues in a computing system. The method also includes selectingfrom the plurality of code modules to generate a first set of codemodules based on a corresponding value metric. The corresponding valuemetric for each code module in the first set of code modules is higherthan a predetermined threshold. The method further includes samplingfrom a remainder of the plurality of code modules unselected for thefirst set of code modules to generate a second set of code modules. Themethod includes running the first set of code modules and the second setof code modules to detect the one or more issues and updating thecorresponding value metric for at least one code module in the pluralityof code modules.

In another form, an apparatus comprising a network interface and aprocessor is provided. The network interface is configured tocommunicate with a plurality of computing devices. The processor iscoupled to the network interface, and configured to obtain a pluralityof code modules configured to detect one or more issues in a computingsystem. The processor is also configured to select from the plurality ofcode modules to generate a first set of code modules based on acorresponding value metric. The corresponding value metric for each codemodule in the first set of code modules is higher than a predeterminedthreshold. The processor is further configured to sample from aremainder of the plurality of code modules unselected for the first setof code modules to generate a second set of code modules. The processoris configured to run the first set of code modules and the second set ofcode modules to detect the one or more issues. The processor is alsoconfigured to update the corresponding value metric for at least onemode module in the plurality of code modules.

In still another form, a non-transitory computer readable storage mediais provided that is encoded with instructions that, when executed by aprocessor, cause the processor to obtain a plurality of code modulesconfigured to detect one or more issues in a computing system. Theinstructions cause the processor to select from the plurality of codemodules to generate a first set of code modules based on a correspondingvalue metric. The corresponding value metric for each code module in thefirst set of code modules is higher than a predetermined threshold. Theinstructions also cause the processor to sample from a remainder of theplurality of code modules unselected for the first set of code modulesto generate a second set of code modules. The instructions further causethe processor to run the first set of code modules and the second set ofcode modules to detect the one or more issues. The instructions alsocause the processor to update the corresponding value metric for atleast one ode module in the plurality of code modules.

One or more advantages described herein are not meant to suggest thatany one of the embodiments described herein necessarily provides all ofthe described advantages or that all the embodiments of the presentdisclosure necessarily provide any one of the described advantages.Numerous other changes, substitutions, variations, alterations, and/ormodifications may be ascertained to one skilled in the art and it isintended that the present disclosure encompass all such changes,substitutions, variations, alterations, and/or modifications as fallingwithin the scope of the appended claims.

1. A method comprising: obtaining a plurality of code modules configuredto detect one or more issues in a computing system, wherein each codemodule of the plurality of code modules is configured to detect acorresponding issue of the one or more issues; selecting from theplurality of code modules to generate a first set of code modules basedon a corresponding value metric that includes a first probability ofdetecting the one or more issues in the computing system and a secondprobability that the one or more issues will be resolved after beingdetected, the corresponding value metric for each code module in thefirst set of code modules being higher than a predetermined threshold;sampling from a remainder of the plurality of code modules unselectedfor the first set of code modules to generate a second set of codemodules that contains fewer code modules than the remainder of theplurality of code modules unselected for the first set of code modules;running the first set of code modules and the second set of code modulesto detect the one or more issues; and updating the corresponding valuemetric for at least one code module in the plurality of code modules. 2.The method of claim 1, wherein the corresponding value metric of aparticular code module is based on one or more of: a history of theparticular code module being run, a history of the particular codemodule detecting the one or more issues, a history of the one or moreissues being corrected after the particular code module detected the oneor more issues, or an impact of the one or more issues the particularcode module is configured to detect.
 3. The method of claim 1, furthercomprising removing at least one legacy code module from the pluralityof code modules based on the corresponding value metric for the at leastone legacy code module falling below another predetermined threshold. 4.The method of claim 1, wherein updating the corresponding value metriccomprises increasing the corresponding value metric by an accumulationmetric for each code module of the plurality of code modules that wasnot run in the first set of code modules or the second set of codemodules.
 5. The method of claim 4, wherein sampling from the remainderof the plurality of code modules unselected for the first set of codemodules comprises selecting at least one unused code module forinclusion in the second set of code modules based on the accumulationmetric corresponding to the at least one unused code module.
 6. Themethod of claim 1, further comprising providing a message to an operatorof the computing system, the message indicating at least one detectedissue that was detected from running the first set of code modules andthe second set of code modules.
 7. The method of claim 6, furthercomprising: determining whether at least one detected issue was resolvedin the computing system; and responsive to a determination that the atleast one detected issue is unresolved in the computing system, loweringthe corresponding value metric of a particular code module that isconfigured to detect the at least one detected issue.
 8. An apparatuscomprising: a network interface configured to communicate with aplurality of computing devices; and a processor coupled to the networkinterface, the processor configured to: obtain a plurality of codemodules configured to detect one or more issues in a computing system,wherein each code module of the plurality of code modules is configuredto detect a corresponding issue of the one or more issues; select fromthe plurality of code modules to generate a first set of code modulesbased on a corresponding value metric that includes a first probabilityof detecting the one or more issues in the computing system and a secondprobability that the one or more issues will be resolved after beingdetected, the corresponding value metric for each code module in thefirst set of code modules being higher than a predetermined threshold;sample from a remainder of the plurality of code modules unselected forthe first set of code modules to generate a second set of code modulesthat contains fewer code modules than the remainder of the plurality ofcode modules unselected for the first set of code modules; run the firstset of code modules and the second set of code modules to detect the oneor more issues; and update the corresponding value metric for at leastone code module in the plurality of code modules.
 9. The apparatus ofclaim 8, wherein the processor is further configured to remove at leastone legacy code module from the plurality of code modules based on thecorresponding value metric for the at least one legacy code modulefalling below another predetermined threshold.
 10. The apparatus ofclaim 8, wherein the processor is configured to update the correspondingvalue metric by increasing the corresponding value metric by anaccumulation metric for each code module of the plurality of codemodules that was not run in the first set of code modules or the secondset of code modules.
 11. The apparatus of claim 10, wherein theprocessor is configured to sample from the remainder of the plurality ofcode modules unselected for the first set of code modules by selectingat least one unused code module for inclusion in the second set of codemodules based on the accumulation metric corresponding to the at leastone unused code module.
 12. The apparatus of claim 8, wherein theprocessor is further configured to cause the network interface toprovide a message to an operator of the computing system, the messageindicating at least one detected issue that was detected from runningthe first set of code modules and the second set of code modules. 13.The apparatus of claim 12, wherein the processor is further configuredto: determine whether the at least one detected issue was resolved inthe computing system; and responsive to a determination that the atleast one detected issue is unresolved in the computing system, lowerthe corresponding value metric of a particular code module that isconfigured to detect the at least one detected issue.
 14. One or morenon-transitory computer readable storage media encoded with softwarecomprising computer executable instructions and, when the software isexecuted, it is operable to cause a processor to: obtain a plurality ofcode modules configured to detect one or more issues in a computingsystem, wherein each code module of the plurality of code modules isconfigured to detect a corresponding issue of the one or more issues;select from the plurality of code modules to generate a first set ofcode modules based on a corresponding value metric that includes a firstprobability of detecting the one or more issues in the computing systemand a second probability that the one or more issues will be resolvedafter being detected, the corresponding value metric for each codemodule in the first set of code modules being higher than apredetermined threshold; sample from a remainder of the plurality ofcode modules unselected for the first set of code modules to generate asecond set of code modules that contains fewer code modules than theremainder of the plurality of code modules unselected for the first setof code modules; run the first set of code modules and the second set ofcode modules to detect the one or more issues; and update thecorresponding value metric for at least one code module in the pluralityof code modules.
 15. The one or more non-transitory computer readablestorage media of claim 14, wherein the corresponding value metric of aparticular code module is based on one or more of a history of theparticular code module being run, a history of the particular codemodule detecting the one or more issues, a history of the one or moreissues being corrected after the particular code module detected the oneor more issues, or an impact of the one or more issues the particularcode module is configured to detect.
 16. The one or more non-transitorycomputer readable storage media of claim 14, wherein the software isfurther operable to cause the processor to remove at least one legacycode module from the plurality of code modules based on thecorresponding value metric for the at least one legacy code modulefalling below another predetermined threshold.
 17. The one or morenon-transitory computer readable storage media of claim 14, wherein thesoftware is further operable to cause the processor to update thecorresponding value metric by increasing the corresponding value metricby an accumulation metric for each code module of the plurality of codemodules that was not run in the first set of code modules or the secondset of code modules.
 18. The one or more non-transitory computerreadable storage media of claim 17, wherein the software is furtheroperable to cause the processor to sample from the remainder of theplurality of code modules unselected for the first set of code modulesby selecting at least one unused code module for inclusion in the secondset of code modules based on the accumulation metric corresponding tothe at least one unused code module.
 19. The one or more non-transitorycomputer readable storage media of claim 14, wherein the software isfurther operable to cause the processor to provide a message to anoperator of the computing system, the message indicating at least onedetected issue that was detected from running the first set of codemodules and the second set of code modules.
 20. The one or morenon-transitory computer readable storage media of claim 19, wherein thesoftware is further operable to cause the processor to: determinewhether the at least one detected issue was resolved in the computingsystem; and responsive to a determination that at least one detectedissue is unresolved in the computing system, lower the correspondingvalue metric of a particular code module that is configured to detectthe at least one detected issue.